Effects of Early Warning Emails on Student Performance

class: title-slide

<br>
<br>
# Effects of Early Warning Emails on Student Performance <br>

.padding_left.pull-down.white[
.bold[_J. Klenke_], T. Massing, N. Reckmann, J. Langerbein, B. Otto, M. Goedicke, C. Hanck
<br>
<br>
<br>
`\(15^{TH}\)` International Conference on Computer Supported Education

Prague, 21-23 April, 2023
]

---
name: course
# Research Idea and Course Description

## Research question

.blockquote[
Does objective and motivating feedback through a warning email have a positive impact on student's performance
]

- Analyzed Course: _Inferential Statistics_ at the University of Duisburg-Essen
- Compulsory for business and economics
  - Weekly 2-hour lecture 
  - Weekly 2-hour exercise
  - We also have other interventions
- __802__ students at the beginning of the semester
  - __337__ students took an exam at the end of the semester

---
# Treatment Assignment
<br>
<br>
- A logit model was used to predict students' probability to pass the exam based on the first 3 online tests
  - The model was trained with the latest data obtained from the previous edition of the same course

- If predicted probability to pass `\(\leq 0.4\)` the student got a warning mail

???

- [Kahoot!](https://kahoot.com/) games used during classes
- Homework (formative assessment) and 5 online tests (summative assessment) on the e-assessment platform  [JACK](https://s3.paluno.uni-due.de/en/forschung/spalte1/e-learning-und-e-assessment)

---
# Course Timeline Main Events

<br>

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#plots/timeline_plot.png" alt="Timeline for the key events in the 2019 summer term course Inferential Statistics (treatment cohort)" width="80%" />
<p class="caption">Timeline for the key events in the 2019 summer term course Inferential Statistics (treatment cohort)</p>
</div>

- The shaded area indicates the period after treatment
- 57 days between the warning mail and `\(1^{st}\)` exam 
- 113 days between the warning email and `\(2^{nd}\)` exam

---
name: literature
# Literature on Warning Systems in Education

.font80[
- <a id='cite-Arnold2012'></a><a href='#bib-Arnold2012'>Arnold and Pistilli (2012)</a> investigated the effect of the signal light system at Purdue University and found a positive effect on student grades
- <a id='cite-baneres2020'></a><a href='#bib-baneres2020'>Bañeres, Rodríguez, Guerrero-Roldán, and Karadeniz (2020)</a> implemented an early warning system but did not analyze the effect on students' performance
- <a id='cite-csahin2019'></a><a href='#bib-csahin2019'>Şahin and Yurdugül (2019)</a> invented an _Intelligent Intervention System_ where students get feedback for each assessment
  - Students emphasized the usefulness of the system 
- <a id='cite-Iver2019'></a><a href='#bib-Iver2019'>Mac Iver, Stein, Davis, Balfanz, and Fox (2019)</a> could not find an effect from their early warning system in the ninth grade
- <a id='cite-Edmunds2002'></a><a href='#bib-Edmunds2002'>Edmunds and Tancock (2002)</a> analyzed the effects of incentives on third and four-graders' reading motivation and did not find an effect
]

--
<br>
.blockquote[
- The literature on the effects of warning system is inconclusive
- Many studies analyzed the system with questionnaires

.padding_left_2[<svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#004c93;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 8c137 0 248 111 248 248S393 504 256 504 8 393 8 256 119 8 256 8zm-28.9 143.6l75.5 72.4H120c-13.3 0-24 10.7-24 24v16c0 13.3 10.7 24 24 24h182.6l-75.5 72.4c-9.7 9.3-9.9 24.8-.4 34.3l11 10.9c9.4 9.4 24.6 9.4 33.9 0L404.3 273c9.4-9.4 9.4-24.6 0-33.9L271.6 106.3c-9.4-9.4-24.6-9.4-33.9 0l-11 10.9c-9.5 9.6-9.3 25.1.4 34.4z"></path></svg> We try to measure the direct effect on students' performance]

]

???

just short -> inconclusive main Point

---
name: RDD
# RDD Toy Example &#8212; I

## Parametric Estimation

<br>

---
# RDD Toy Example &#8212; II

## Non-parametric Estimation

- We used the data-driven approach by <a id='cite-imbensoptimal2009'></a><a href='#bib-imbensoptimal2009'>Imbens and Kalyanaraman (2009)</a> to determine the bandwidth

???
- The method fits the bandwidth as widely as possible without introducing other confounding effects

---
# Model Assumptions

- The running variable `\(W\)` (predicted probability to pass the exam) must not have a jump around the cutoff in the density function

.pull-left-2[
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#plots/test_cont_label.png" alt="Graphical illustration of the McCrary sorting test" width="80%" />
<p class="caption">Graphical illustration of the McCrary sorting test</p>
</div>
]

.pull-right-1[
<br>

- There is no jump in the density around the cutoff point of `\(0.4\)`
- `\(p\)`-value: `\(0.509\)`
- The incentive to manipulate the treatment is quite low

]

.pull-down[
- Also, standard IV estimation assumptions must hold
]

---
name: results
# Empirical Results &#8212; I

.pull-left-2[
<br>
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#plots/model_plot_label.png" alt="Graphical illustration of the RDD model" width="95%" />
<p class="caption">Graphical illustration of the RDD model</p>
</div>

]
--

.pull-right-1[
### Estimate 
- LATE: 0.193 
  - SE: 4.889
  - `\(p\)`-value: 0.968
- Bandwidth: 0.255
- `\(N\)`: 126

]

???

- theoretisch sollten Sie jetzt einen großen Sprung sehen
- covariates auch getestet
- other approaches were used

---

# Empirical Results &#8212; II

- The LATE estimate is positive but not significant  
  - An estimate of `\(0.193\)` means that students who received the warning email achieved `\(0.193\)` points more than comparable students who did not 
  - Compared to the `\(60\)`-point exam, the effect size seems limited

- Bandwidth of `\(0.255\)`
  - Only students with a predicted probability `\(0.4\)` (cutoff) `\(\pm \ 0.255\)` (bandwidth), are included in the analysis
  
--

- This leads to the effective sample size of `\(126\)` students

---
name: discussion
# Discussion &#8212; I

- Our RDD results do not provide evidence that the warning email has a significant effect on students' results (or behavior)
- The variance around the cutoff is relatively high, which compromises the detection of an effect 
- Many individuals are not included in the final analysis for several reasons
  - Students dropping the course
  - Students far away from the cutoff are not providing much information

.padding_left_2[<svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#004c93;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 8c137 0 248 111 248 248S393 504 256 504 8 393 8 256 119 8 256 8zm-28.9 143.6l75.5 72.4H120c-13.3 0-24 10.7-24 24v16c0 13.3 10.7 24 24 24h182.6l-75.5 72.4c-9.7 9.3-9.9 24.8-.4 34.3l11 10.9c9.4 9.4 24.6 9.4 33.9 0L404.3 273c9.4-9.4 9.4-24.6 0-33.9L271.6 106.3c-9.4-9.4-24.6-9.4-33.9 0l-11 10.9c-9.5 9.6-9.3 25.1.4 34.4z"></path></svg>  Thus precise estimation of the treatment becomes more difficult]

---
# Discussion &#8212; II

- Students also get feedback through their online tests
- The warning may also lead weak students to postpone participation to a later semester
  - The cost in our program to postpone exams is quite low
- The objective feedback and motivation from one warning email is rather small

---
name: f_reaserach
# Further Research

- The effect on the dropout rate from such warning emails or systems requires further attention 
- An automatic repeated feedback system could have a more significant impact on student's motivation
  - Detailed recurring feedback could also be used to guide students

<br>
.blockquote[
We see the open and transparent communication of the student's performance to the students as a positive aspect of the system
]

---
name: references
# References

.font80[

Arnold, K. E. and M. Pistilli (2012). "Course signals at Purdue: using learning analytics to increase student success". Eng. In: _ACM
International Conference Proceeding Series_. LAK '12. ACM, pp. 267-270.

Bañeres, D., M. E. Rodríguez, A. E. Guerrero-Roldán, et al. (2020). "An Early Warning System to Detect At-Risk Students in Online Higher
Education". In: _Applied Sciences_ 10.13, p. 4427.

Edmunds, K. and S. M. Tancock (2002). "Incentives: The effects on the reading motivation of fourth‐grade students". In: _Reading Research and
Instruction_ 42.2, pp. 17-37.

Imbens, G. and K. Kalyanaraman (2009). "Optimal Bandwidth Choice for the Regression Discontinuity Estimator". In: _National Bureau of Economic
Research_ 1.14726.

Mac Iver, M. A., M. L. Stein, M. H. Davis, et al. (2019). "An Efficacy Study of a Ninth-Grade Early Warning Indicator Intervention". In: _Journal
of Research on Educational Effectiveness_ 12.3, pp. 363-390.

Şahin, M. and H. Yurdugül (2019). "An intervention engine design and development based on learning analytics: the intelligent intervention system
(In 2 S)". In: _Smart Learning Environments_ 6.1, p. 18.
]